Display Tracking Systems and Methods

ABSTRACT

A tracked device may be used in an extended reality system in coordination with a tracking device. The tracked device may be ordinarily difficult to track, for example due to changing appearances or relatively small surface areas of unchanging features, as may be the case with an electronic device with a relatively large display surrounded by a thin physical outer boundary. In these cases, the tracked device may periodically present an image to the tracking device that the tracking device stores as an indication to permit tracking of a known, unchanging feature despite the image not being presented continuously on the display of the tracked device. The image may include a static image, designated tracking data overlaid on an image frame otherwise scheduled for presentation, or extracted image features from the image frame otherwise scheduled for presentation. Additional power saving methods and known marker generation methods are also described.

BACKGROUND

The present disclosure relates generally to image feature tracking and,more particularly, to using image data processing to determine and tracka location of an electronic device.

This section is intended to introduce the reader to various aspects ofart that may be related to various aspects of the present disclosure,which are described and/or claimed below. This discussion is believed tobe helpful in providing the reader with background information tofacilitate a better understanding of the various aspects of the presentdisclosure. Accordingly, it should be understood that these statementsare to be read in this light, and not as admissions of prior art.

Extended reality (XR), virtual reality (VR), augmented reality (AR), andmixed reality (MR) devices may provide an immersive experience in avirtual world. To allow a person to interact with virtual objects in thevirtual world, some of these devices may use a camera to track alocation of an object in the real world. Based on the location of theobject in the real world, the virtual objects may behave in a particularway. For example, a person may use their hand in the real world to“push” or “pull” virtual objects in the virtual world. Tracking thelocation of an object based on its shape works well when the shape ofthe object remains constant or changes sufficiently slowly. Yet objectsthat display images on electronic displays, such as electronic deviceslike handheld or wearable devices, may be more difficult to track.Indeed, the apparent location of these objects could change depending onthe images that happen to be displayed on their displays, potentiallydisrupting the experience of a seamless interaction between the realworld and the virtual world.

SUMMARY

A summary of certain embodiments disclosed herein is set forth below. Itshould be understood that these aspects are presented merely to providethe reader with a brief summary of these certain embodiments and thatthese aspects are not intended to limit the scope of this disclosure.Indeed, this disclosure may encompass a variety of aspects that may notbe set forth below.

In a first example, two electronic devices may be located in a physicalenvironment. The first electronic device may present an image frame withtracking data on its display. The second electronic device may include acamera subsystem, memory, and a processor that monitors the trackingdata as a way to track a location the display of the first electronicdevice, and thus a location of the first electronic device. Afteridentifying the location of the first electronic device, the secondelectronic device may generate an animation, an immersive experience, animage, or the like, that follows or aligns with the location of thefirst electronic device. An animation, an immersive experience, or animage that tracks the location of the first electronic device in thephysical environment may be overlaid or rendered adjacent on the firstelectronic device in a virtual environment. As the first electronicdevice moves within the physical environment, images of the firstelectronic device and of the animation, the immersive experience, or theimage in the virtual environment move in association with the firstelectronic device as its location changes.

In a second example, a method may include using an early warningindication, or a signal indication, of incoming tracking data as a wayto trigger returning power to certain portions of an electronic device.The electronic device may be a tracking device used to capture images ofa tracked electronic device. The images may be presented via a displayof the electronic device. By tracking the tracking data presented on adisplay of the tracked electronic device, the electronic device maytrack a location of the tracked electronic device. The electronic devicemay use the location of the tracked electronic device to manipulate ananimation, such that the animation remains overlaid or a particulardistance from the tracked electronic device in a virtual environmentconveyed via the presentation of the images even while the trackedelectronic device moves in the physical environment.

In a third example, a tracked device may generate its own tracking data.Tracking data may be generated by blending tracking data with image datato be presented on the tracked device. A tracking device may use thetracking data to determine where the tracked device is located within aphysical environment, where a location of the tracked device may becorrelated to a location of the tracking data in a captured image of thephysical environment. Once located, images of the tracked device in avirtual world may be displayed proximate to or in association with othervirtual images rendered by the tracking device. The virtual images maybe associated with an animation, an immersive experience, an image, orthe like.

Various refinements of the features noted above may exist in relation tovarious aspects of the present disclosure. Further features may also beincorporated in these various aspects as well. These refinements andadditional features may exist individually or in any combination. Forinstance, various features discussed below in relation to one or more ofthe illustrated embodiments may be incorporated into any of theabove-described aspects of the present disclosure alone or in anycombination. The brief summary presented above is intended only tofamiliarize the reader with certain aspects and contexts of embodimentsof the present disclosure without limitation to the claimed subjectmatter.

BRIEF DESCRIPTION OF THE DRAWINGS

Various aspects of this disclosure may be better understood upon readingthe following detailed description and upon reference to the drawings inwhich:

FIG. 1 is a block diagram of an electronic device with an electronicdisplay, in accordance with an embodiment;

FIG. 2 is an example of the electronic device of FIG. 1 , in accordancewith an embodiment;

FIG. 3 is another example of the electronic device of FIG. 1 , inaccordance with an embodiment;

FIG. 4 is another example of the electronic device of FIG. 1 , inaccordance with an embodiment;

FIG. 5 is another example of the electronic device of FIG. 1 , inaccordance with an embodiment;

FIG. 6 is a diagrammatic representation of a tracking device tracking atracked device, in accordance with an embodiment;

FIG. 7 is a diagrammatic representation of the tracked device of FIG. 6presenting image frames via the display of FIG. 1 over time, inaccordance with an embodiment;

FIG. 8 is a diagrammatic representation of operations and block diagramsof the tracked device of FIG. 6 and the tracking device of FIG. 6 overtime, in accordance with an embodiment;

FIG. 9 is a block diagram of the tracked device of FIG. 6 and thetracking device of FIG. 6 , in accordance with an embodiment;

FIG. 10 is a flow diagram of a process for operating the tracked deviceof FIG. 6 during a tracking operation corresponding to the trackingdevice of FIG. 6 , in accordance with an embodiment;

FIG. 11 is a flow diagram of a process for operating the tracking deviceof FIG. 6 during a tracking operation corresponding to the trackeddevice of FIG. 6 , in accordance with an embodiment;

FIG. 12 is a diagrammatic representation of the tracked device of FIG. 6and the tracking device of FIG. 6 exchanging an image data sample, inaccordance with an embodiment;

FIG. 13 is a block diagram of the tracking device of FIG. 6 and of thetracked device of FIG. 6 , which includes a subsampler, in accordancewith an embodiment;

FIG. 14 is a block diagram of the tracking device of FIG. 6 and of thetracked device of FIG. 6 , which includes a feature extractor, inaccordance with an embodiment; and

FIG. 15 is a diagrammatic representation of the tracking device of FIG.6 superimposing or overlaying image data generated by its own imageprocessing circuitry on captured image data that depicts the trackeddevice of FIG. 6 , in accordance with an embodiment;

FIG. 16 is a block diagram of a tracked device and a tracking devicethat exchange first and second tracking data, in accordance with anembodiment; and

FIG. 17 is a flow diagram of a process for operating the tracking deviceof FIG. 16 during a tracking operation corresponding to the trackeddevice of FIG. 16 , in accordance with an embodiment.

DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS

One or more specific embodiments will be described below. In an effortto provide a concise description of these embodiments, not all featuresof an actual implementation are described in the specification. Itshould be appreciated that in the development of any such actualimplementation, as in any engineering or design project, numerousimplementation-specific decisions are made to achieve the developers'specific goals, such as compliance with system-related andbusiness-related constraints, which may vary from one implementation toanother. Moreover, it should be appreciated that such a developmenteffort might be complex and time consuming, but would nevertheless be aroutine undertaking of design, fabrication, and manufacture for those ofordinary skill having the benefit of this disclosure.

When introducing elements of various embodiments of the presentdisclosure, the articles “a,” “an,” and “the” are intended to mean thatthere are one or more of the elements. The terms “comprising,”“including,” and “having” are intended to be inclusive and mean thatthere may be additional elements other than the listed elements.Additionally, it should be understood that references to “oneembodiment” or “an embodiment” of the present disclosure are notintended to be interpreted as excluding the existence of additionalembodiments that also incorporate the recited features. Furthermore, thephrase A “based on” B is intended to mean that A is at least partiallybased on B. Moreover, the term “or” is intended to be inclusive (e.g.,logical OR) and not exclusive (e.g., logical XOR). In other words, thephrase A “or” B is intended to mean A, B, or both A and B.

By way of example, an electronic display may be or represent one or moredisplays for an entertainment system, such as in an extended reality(XR) system. An extended reality (XR) system may includereal-and-virtual combined environments and human-machine interactionsgenerated by computer technology and wearables. An extended reality (XR)system may support augmented reality (AR), mixed reality (MR), andvirtual reality (VR). In some instances, the XR system may enablesimulation of a three-dimensional image or environment that can beinteracted with in a seemingly real or physical way. Sometimes thisinvolves using special electronic equipment (e.g., helmet or glasseswith a display inside or gloves fitted with sensors), referred to asvirtual reality (VR). The XR system may also enable thethree-dimensional image to project onto real objects, merging realities,referred to as augmented reality (AR). In other instances, theentertainment system may combine elements of both AR and VR, in whichreal-world and digital objects interact, referred to as mixed reality(MR). The XR system may also update the displayed image as the usermoves or changes orientation using gyroscopes, accelerometers, or thelike. In many cases, the image data to be displayed for the XR systemmay include multiple frames of image data (e.g., streams of image data).

An XR system may take many forms. For example, an XR system may take theform of a handheld or tablet device that uses its display to provide anaugmented reality (AR) or mixed reality (MR) experience. Other XRsystems may take the form of head-mounted displays (HMDs). In these, theuser may see two separate images of image data on the display of the XRsystem, one for the left eye and one for the right eye, which the usermay perceive as a single image in three-dimensional form. Moreover, theXR system may include multiple cameras capturing the multiple frames ofimage data, and may use multiple devices and/or chips for rendering anddisplaying the image data.

Keeping the foregoing in mind, an XR system may project a generatedimage in association with a location of one or more virtual objectsand/or one or more real objects. Indeed, the XR system may generate atwo-dimensional or three-dimensional image for projection on or inassociation with real objects (e.g., AR system), virtual objects (e.g.,VR system), or a combination of the real and virtual objects (e.g., MRsystem). When projecting the image, the XR system may project an imageat a location relatively defined to a location of the one or more realobjects and/or the one or more virtual objects. Thus, when the realobjects or virtual objects, or both, move, the projected image istracked with the real objects and/or virtual objects. The trackedprojected image is perceivable to the user viewing the rendered imagesof the XR system without simultaneously being perceivable to athird-party viewing the real objects without the XR system.

Tracking a real object with a relatively fixed appearance in a real orvirtual environment may be more straightforward than tracking an objectthat is changing. Difficulties in tracking a real object with a variableappearance may be exacerbated when tracking the real object while thereal object is moving within the real environment. Even more so, it maybe difficult to track an object that continues to display differentimages on its display. To overcome this difficulty, the XR system maytrack specific patterns on an electronic display of an electronic devicebeing used to present a video stream.

As such, the present disclosure describes systems and methods forefficiently tracking an electronic display of an electronic device whilethe electronic display of the electronic device is presenting rapidlychanging images, a video stream, or static images. Specifically, inaddition to physical boundaries of the electronic device, an XR systemmay watch for and follow image frames of image data on the electronicdisplay of the electronic device that include designated tracking data,expected static image frames, or expected defined image features in animage or video stream of the electronic device. The designated trackingdata may be a watermark (e.g., watermark image), a pattern (e.g.,patterned image, a high-contrast pattern, a black—white pattern, amulti-color pattern), or the like to be overlaid on an image frame to bepresented. Image characteristics of the designated tracking data, andthus of the watermark or pattern, may be adjusted dynamically based onambient brightness level, and thus may be presented differently when theelectronic display is outside in a bright environment or inside in adimmer environment. A static image frame may be an unchanging imagepresented on the electronic display. The defined image features may beselected subsets of an image frame within the image, such as ahigh-contrast region of an image frame, a high-contrast font, any font,any region of an image frame a threshold amount different from eachother region of the image frame (e.g., 50%, 60%, 70%, or any differencein average brightness or average color value), a region of a largeamount of color, or the like. The electronic device may transmit anindication to the XR system, or the XR system may already store anindication, of the defined image features or of the static image, topermit the XR system to readily track the electronic display.

Each of these methods (e.g., designated tracking data, static imageframes, defined image features) may be thought of as creating oridentifying a trackable feature on the electronic display. The XR systemtracking the trackable feature of the electronic display may increasethe accuracy of tracking the electronic display while reducingperceivable delays or errors associated with tracking the electronicdisplay even when the electronic display presents image data. For easeof discussion, the term “tracked device” may refer to an electronicdevice (e.g., a handheld device) that presents the trackable feature onits electronic display and the term “tracking device” may be refer to anelectronic device (e.g., an XR device) that tracks the trackablefeature. The tracking device may also use the location of the trackeddevice to perform another operation. For example, the tracking devicemay use the location of the tracked device to render a three-dimensionalimage that interacts with the electronic display, the tracked device, orthe trackable feature and/or that is located relative to the trackablefeature (e.g., a defined distance from the trackable feature). In thisway, the tracking device may be an XR system, or may be differentsuitable system, such as another electronic device running an extendedreality (XR) application. The XR systems and application may be used asan entertainment system, but may also have a wide range of applicabilityin a variety of consumer and business products, such as healthcare,employee training, and the like. Thus, omission of specific discussionof these other use cases should not be construed as intentionallylimiting or restrictive.

A signal may be transmitted from a first, tracked device to a second,tracking device (e.g., an XR system) to provide a warning for when thedesignated tracking data or defined image features are upcoming fordisplay on the electronic display of the tracked device. In response tothe signal, the tracking device may power on or increase power suppliedto its camera subsystem and/or processing subsystem, as to prepare tocapture an image of the tracked device's electronic display at a timethat the designated tracking data or defined image features arepresented. After capturing the image data, the tracking device mayreduce power to or power off the camera subsystem and/or processingsubsystem, which may reduce power consumption of the tracking device.

In some cases, the tracking device may adjust an amount of powersupplied to the camera subsystem and/or processing subsystem in responseto a duration of time passing from a previous tracking data presentationtime, where the duration of time may be a mutually followed delay ortrigger time by which both the tracked device and the tracking deviceare synchronized. In this way, a processor of the tracking device may,in response to a duration of time passing from a previous tracking datapresentation time, increase an amount of power supplied to the camerasubsystem, to a processing subsystem of the second electronic device, orboth.

Furthermore, the tracking device may filter out designated tracking datafrom a captured video stream of the tracked device. Although the trackeddevice may display certain designated tracking data, which could benoticeable to a person looking at the display, the tracking device mayfilter out the tracking data when it is shown on a display of thetracking device. For example, a tracking pattern may appear in the realworld on the electronic display of the tracked device. However, in an ARor MR environment generated by the tracking device, the tracking patternmay be filtered out and thus may not be visible on the tracked device inthe virtual world.

Moreover, the tracking device may use motion sensing data from aninertial measurement unit (IMU) of the tracked device, its own IMU, orboth to predict where within an image frame region captured by thecamera of the tracking device that an electronic display of the trackeddevice is expected to appear at a future time. Predicting where thetracked device is expected to be at a future time may reduce the timeinvolved in finding where the electronic display is located. Forexample, predicting which image frame region in which the electronicdisplay of the tracked device will appear may reduce the time otherwiseused for searching the entire image frame.

With this in mind, an electronic device 10 including a display 12 (e.g.,display device) is shown in FIG. 1 . As is described in more detailbelow, the electronic device 10 may be any suitable electronic device,such as a computer, a mobile phone, a portable media device, a tablet, atelevision, a virtual-reality headset (e.g., an extended realityheadset), a vehicle dashboard, and the like. Thus, it should be notedthat FIG. 1 is merely one example of a particular implementation and isintended to illustrate the types of components that may be present in anelectronic device 10.

The display 12 may be any suitable electronic display. For example, thedisplay 12 may include a self-emissive pixel array having an array ofone or more of self-emissive pixels. The display 12 may include anysuitable circuitry to drive the self-emissive pixels, including forexample row driver and/or column drivers (e.g., display drivers). Eachof the self-emissive pixel may include any suitable light emittingelement, such as a LED, one example of which is an OLED. However, anyother suitable type of pixel, including non-self-emissive pixels (e.g.,liquid crystal as used in liquid crystal displays (LCDs), digitalmicromirror devices (DMD) used in DMD displays) may also be used.

In the depicted embodiment, the electronic device 10 includes thedisplay 12, one or more input devices 14, one or more input/output (I/O)ports 16, a processor core complex 18 having one or more processor(s) orprocessor cores, local memory 20, a main memory storage device 22, anetwork interface 24, a power source 26 (e.g., power supply), and imageprocessing circuitry 28. The various components described in FIG. 1 mayinclude hardware elements (e.g., circuitry), software elements (e.g., atangible, non-transitory computer-readable medium storing instructions),or a combination of both hardware and software elements. It should benoted that the various depicted components may be combined into fewercomponents or separated into additional components. For example, thelocal memory 20 and the main memory storage device 22 may be included ina single component. The image processing circuitry 28 (e.g., a graphicsprocessing unit, an image signal processor, a display pipeline) may beincluded in the processor core complex 18.

The processor core complex 18 may execute instruction stored in localmemory 20 and/or the main memory storage device 22 to performoperations, such as generating and/or transmitting image data. As such,the processor core complex 18 may include one or more general purposemicroprocessors, one or more application specific integrated circuits(ASICs), one or more field programmable logic arrays (FPGAs), or anycombination thereof.

In addition to instructions, the local memory 20 and/or the main memorystorage device 22 may store data to be processed by the processor corecomplex 18. Thus, the local memory 20 and/or the main memory storagedevice 22 may include one or more tangible, non-transitory,computer-readable mediums. For example, the local memory 20 may includerandom access memory (RAM) and the main memory storage device 22 mayinclude read-only memory (ROM), rewritable non-volatile memory such asflash memory, hard drives, optical discs, and/or the like.

The network interface 24 may communicate data with another electronicdevice and/or a network. For example, the network interface 24 (e.g., aradio frequency system) may enable the electronic device 10 tocommunicatively couple to a personal area network (PAN), such as aBluetooth network, a local area network (LAN), such as an 1622.11x Wi-Finetwork, and/or a wide area network (WAN), such as a fourth generation(4G), fifth generation (5G), or Long-Term Evolution (LTE) cellularnetwork.

The power source 26 may provide electrical power to one or morecomponents in the electronic device 10, such as the processor corecomplex 18 and/or the display 12. Thus, the power source 26 may includeany suitable source of energy, such as a rechargeable lithium polymer(Li-poly) battery and/or an alternating current (AC) power converter.The I/O ports 16 may enable the electronic device 10 to interface withother electronic devices. For example, when a portable storage device isconnected, the I/O port 16 may enable the processor core complex 18 tocommunicate data with the portable storage device. The input device 14may enable user interaction with the electronic device 10, for example,by receiving user inputs via a button, a keyboard, a mouse, a trackpad,and/or the like. The input device 14 may also include touch-sensingcomponents in the display 12. The touch-sensing components may receiveuser inputs by detecting occurrence and/or position of an objecttouching the surface of the display 12.

In addition to enabling user inputs, the display 12 may include one ormore display panels each having one or more display pixels. Each displaypanel may be a separate display device or one or more display panels maybe combined into a same device. The display 12 may control lightemission from the display pixels to present visual representations ofinformation, such as a graphical user interface (GUI) of an operatingsystem, an application interface, a still image, or video content, bydisplaying frames based on corresponding image data. The display 12 maydisplay frames based on image data generated by the processor corecomplex 18 and/or the image processing circuitry 28. Additionally oralternatively, the display 12 may display frames based on image datareceived via the network interface 24, an input device 14, an I/O port16, or the like.

As described above, the electronic device 10 may be any suitableelectronic device. To help illustrate, an example of the electronicdevice 10, a handheld device 10A, is shown in FIG. 2 . The handhelddevice 10A may be a portable phone, a media player, a personal dataorganizer, a handheld game platform, and/or the like. For illustrativepurposes, the handheld device 10A may be a smart phone, such as anyIPHONE® model available from Apple Inc.

The handheld device 10A includes an enclosure 30 (e.g., housing). Theenclosure 30 may protect interior components from physical damage and/orshield them from electromagnetic interference, such as by surroundingthe display 12. The display 12 may display a graphical user interface(GUI) 32 having an array of icons. When an icon 34 is selected either byan input device 14 or a touch-sensing component of the display 12, anapplication program may launch.

The input devices 14 may be accessed through openings in the enclosure30. The input devices 14 may enable a user to interact with the handhelddevice 10A. For example, the input devices 14 may enable the user toactivate or deactivate the handheld device 10A, navigate a userinterface to a home screen, navigate a user interface to auser-configurable application screen, activate a voice-recognitionfeature, provide volume control, and/or toggle between vibrate and ringmodes. The I/O ports 16 may be accessed through openings in theenclosure 30 and may include, for example, an audio jack to connect toexternal devices.

Another example of a suitable electronic device 10, specifically atablet device 10B, is shown in FIG. 3 . The tablet device 10B may be anyIPAD® model available from Apple Inc. A further example of a suitableelectronic device 10, specifically a computer 10C, is shown in FIG. 4 .For illustrative purposes, the computer 10C may be any MACBOOK® or IMAC®model available from Apple Inc. Another example of a suitable electronicdevice 10, specifically a watch 10D, is shown in FIG. 5 . Forillustrative purposes, the watch 10D may be any APPLE WATCH® modelavailable from Apple Inc. As depicted, the tablet device 10B, thecomputer 10C, and the watch 10D each also includes a display 12, inputdevices 14, I/O ports 16, and an enclosure 30. The display 12 maydisplay a GUI 32. Here, the GUI 32 shows a visualization of a clock.When the visualization is selected either by the input device 14 or atouch-sensing component of the display 12, an application program maylaunch, such as to transition the GUI 32 to presenting the icons 34discussed in FIG. 2 and FIG. 3 .

As described above, the display 12 may be tracked when a part of anextended reality (XR) system, such as to enable another electronicdevice to render on its display an image at a location associated with(e.g., proximate to, equal to, a threshold distance from) the locationof the display 12 in an ambient environment. To elaborate, FIG. 6 is adiagrammatic representation of a tracked device 60 (e.g., a smartphone,as depicted in FIG. 6 ) being tracked by a tracking device 62 (e.g., atablet, as depicted in FIG. 6 ). Both the tracked device 60 and thetracking device 62 are examples of the electronic device 10.

The tracking device 62 may track one or more tangible electronicdevices, objects, people, or the like within a space captured by itscamera subsystem. In the example of FIG. 6 , the tracking device 62tracks both the tracked device 60 and an object 64. The object 64 mayhave a fixed or stationary location (e.g., position, orientation,height) within the real, physical world (e.g., an ambient environment 66surrounding the tracking device 62, the object 64, and the trackeddevice 60). The tracking is represented by dashed lines 68 around therendered images of the tracked device 60 and the object 64 presented ona display 12A of the tracking device 62. It may be easier for thetracking device 62 to track the object 64 than to track the trackeddevice 60 since the object 64 is stationary and has wide distinct edgesof the frame visible to processing circuitry of the tracking device 62.When tracking, the tracking device 62 may use a lock on the location ofa target (e.g., target object, target person, target device) as areference location to overlay or associate a generated three-dimensionalor two-dimensional image. For example, the tracking device 62 overlays agenerated star image 70 on captured images of the object 64 and theambient environment 66 when tracking the object 64, and in an actualimplementation any static or variable image or video stream may beoverlaid as a mixed reality augmentation when tracking, includinganimations, special effects, inset images, data, text, shapes, or thelike. In some embodiments, the static or variable image or video streammay be associated with an accompanying audible signal (e.g., a tone,chime or music) conveyed through the tracking device 62.

Tracking the object 64 may be less complex than tracking the trackeddevice 60. The tracking device 62 may track the object 64 by identifyinga set of prominent features (e.g., edges, angles, colors) of the object64 known a-priori by tracking device 62 within captured image data thatincludes a depiction of the object 64. By using cameras to look at theambient environment 66, the tracking device 62 may identify suchfeatures and determine the location in the ambient environment 66 of theobject 64, such as with respect to the tracking device 62 itself oranother reference point or object. When this tracking operation isperformed continuously, mixed reality augmentations will “stick” verysmoothly on top or around the unchanging object 64 being tracked. Forexample, the star image 70 tracks the object 64 as the tracked device60, the tracking device 62, or both move in space relative to the object64. When the object 64 is tracked, the tracking device 62 generates thestar image 70 to continuously overlay the image of the object 64 evenwhile the image of object 64 moves due to the movement of the trackingdevice 62.

Keeping this in mind, the tracking device 62 may perform more complexoperations to track the tracked device 60. Indeed, the tracked device 60may present variable image data and have less prominent or fewer staticfeatures. Thus, the tracked device 60 may be more difficult to trackwhen compared to tracking the object 64 that has static and prominentfeatures. Indeed, static portions of the tracked device 60 may belimited to an outer physical edge 72 of the tracked device 60, which maybe relatively thin when a display 12B is designed to take up the vastmajority (e.g., 90%) of a front surface of the tracked device 60. Thus,it may be desirable for the tracking device 62 to use systems andmethods that enable reliable tracking of a wide range of tracked devices60 presenting a wide range of image data.

In one such example, represented in FIG. 7 , the tracking device 62 maywatch for a reference image to be presented via the tracked device 62.The reference image may be known to the tracking device 62, and thus maybe used as a static reference by which to determine a location of thetracked device 62 despite other changes in tracked device 62 location orthe images presented.

To elaborate, FIG. 7 is a diagrammatic representation of the trackeddevice 60 presenting image frames via a display 12B over time. Althoughthe image frames change over time, systems and methods described hereinmay also work with static image frames. Indeed, the tracking device 62may monitor the tracked device 60 to watch for designated tracking data82 to appear in captured image data of the ambient environment 66. Thedesignated tracking data 82 may be a watermark, a pattern, or the likethat may be overlaid during image processing operations on an imageframe to be presented by the tracked device 60. In some cases, thedesignated tracking data 82 is any data known by the tracking device 62emitted, conveyed, presented, or the like by the tracked device 60 as tobeckon the tracking device 62 to its location and/or its screen locationwithin a three-dimensional space. The tracking device 62 may use thedesignated tracking data 82 in captured image data to identify wherewithin the captured image data that the display 12B is located, and thuswhere the image data corresponding to the tracked device 60 is within adataset for the captured image data. Once the tracking device 62localizes the tracked device 60 in the dataset, the tracking device 62can localize the tracked device 60 in three-dimensional space (e.g.,ambient environment 66), can convey audio-visual virtual content and/oraugmentations around, on top of, and/or proximate to the tracked device60 based on the mixed reality support from the tracking device 62 oranother device, or other suitable processing operations that benefitfrom knowing the location of the tracked device (e.g., the screen of thetracked device).

Additional synchronization circuitry included in the tracked device 60and the tracking device 62 may help perform these improved trackingoperations, as FIG. 8 helps to illustrate. FIG. 8 is a diagrammaticrepresentation of operations and block diagrams of the tracked device 60and the tracking device 62 over several image frames (e.g., over time(t)). The tracked device 60 includes device tracking primary circuitry94 and the tracking device 62 includes device tracking secondarycircuitry 96. The device tracking primary circuitry 94 and the devicetracking secondary circuitry 96 operate together as synchronizationcircuitry to align presentation of tracking data on the display 12B(e.g., on tracked device 60) with image capture operations of thetracking device 62.

The tracked device 60 also includes a radio controller 98A and thetracking device 62 includes a radio controller 98B. Each radiocontroller 98 may include a control system, and thus a processor (e.g.,processor core complex 18) to execute instructions stored in memory(e.g., memory 20, storage devices 22) to perform operations, such asradio frequency communication transmission and control operations.Indeed, the radio controllers 98 may interface with the I/O ports 16,input devices 14, network interface 24, or the like of the other radiocontroller 98 to communicate data between the tracked device 60 and thetracking device 62. The radio controllers 98 may communicate usingwireless or wired connections and signals.

At a first time and first frame (e.g., “Frame t”), the tracked device 60presents image data on the display 12B according to previously receivedimage data. While presenting a second frame (e.g., “Frame t+1”) and thusat a second time, the device tracking primary circuitry 94 determinedthat a next frame is to include designated tracking data 82. In responseto this determination, the device tracking primary circuitry 94transmits a notification 100 signal to the tracking device 62 via theradio controllers 98. The notification 100 may be any suitable signal,including a pulse, a flag, a bit being set in a register, or the like.The tracking device 62 interprets the notification 100 signal asindicating that a next or incoming image frame includes designatedtracking data 82 and proceeds to notify the device tracking secondarycircuitry 96. The device tracking secondary circuitry 96 may generate acontrol signal to instruct the one or more cameras of a camera subsystem102 and a processing subsystem 104 to make an image capture and processthe image capture. In some embodiments, the one or more cameras ofcamera subsystem 102 may be in a low power or disabled state at orbefore the second time. The device tracking secondary circuitry 96 may,additionally or alternatively, generate a power control signal toinstruct power management circuitry of the power source 26 to recoupleor increase power supplied to the one or more cameras of a camerasubsystem 102 and to a processing subsystem 104.

At a third time when a third frame (e.g., “Frame t+2”) is presented, thetracking device 62 operates the camera subsystem 102 to capture imagedata of the display 12B, of the tracked device 60, and/or of the ambientenvironment 66. Thus, the camera subsystem 102 may capture an image ofthe tracked device 60 while it is displaying the designated trackingdata 82. Captured image data may be transmitted from the camerasubsystem 102 to the processing subsystem 104 as a dataset forprocessing to identify the designated tracking data 82 within thedataset. While the processing subsystem 104 processes the captured imagedata, the tracked device 60 may proceed to prepare to update the display12B with a next image frame. In this case, the next image frame is arepeated image frame without the designated tracking data 82. Theprocessing subsystem 104 may determine a location 106 of where thedisplay 12B is located in the ambient environment before, at, or afterthe next image frame being presented at the fourth time.

The processing subsystem 104 may use the location to adjust virtualcontent in relation to the tracked device 60 via the display 12A. Insome cases, the processing subsystem 104 may filter out the designatedtracking data 82 from the image frames to be presented on the display12A of the tracking device 62. Filtering out the designated trackingdata 82 from the image frames to be presented on the display 12A of thetracking device 62 may reduce or eliminate a likelihood that thedesignated tracking data 82 is perceivable by a person viewing thedisplay 12A of the tracking device 62.

In some embodiments, conveyance of designated tracking data 82 ondisplay 12B of tracked device 60 may include obfuscation of some or allaudio-visual content conveyed by tracked device 60. For example, afterlocation 106 is determined by tracking device 62, tracked device 60 mayenter a state where at least a portion of audio-visual information ontracked device 60 (e.g., on display 12B) ceases to be conveyed so thatother users in ambient environment 66 cannot observe that portion of theaudio-visual information. In this example, a full set of information ondisplay 12B may not be visible without viewing display 12B throughdisplay 12A of tracking device 62. In some embodiments, tracked display12B remains in an obfuscated state until tracking device 62 discontinuestracking tracked device 60. In some embodiments, tracked device 60and/or display 12B transitions to a low power, inactive, sleep and/orother state where audio-visual information ceases to be conveyed ontracked device 60 (e.g., display 12B may turn off).

Elaborating further on the device tracking primary circuitry 94 anddevice tracking secondary circuitry 96, FIG. 9 is a block diagram of thetracked device 60 and the tracking device 62. In addition to the blocksdescribed in FIG. 8 , the tracked device 60 may also include a graphicsprocessing pipeline 118 and a display processing pipeline 120. Thedevice tracking primary circuitry 94 may include a tracking datagenerator 122, blender 124, and a pulse generator 126. The pulsegenerator 126 may output a notification 100 to a delay block 130 and theradio controller 98A. Furthermore, in addition to the blocks describedin FIG. 8 , the tracking device 62 may also include a pulse decoder 132,a camera shutter controller 134, and an interrupt generator 136 as partof the device tracking secondary circuitry 96.

The tracked device 60 may generate image data for presentation via thedisplay 12B using the graphics processing pipeline 118. The generatedimage data may be adjusted and prepared for presentation by the displayprocessing pipeline 120. The display processing pipeline 120 may includeone or more sequentially coupled circuits and/or sequentially performedoperations to adjust one or more portions of the image data sequentiallyprior to output to the display 12B. For example, a first portion of theimage data may be adjusted using a first operation. The second portionmay be adjusted using the first operation afterward. While the secondportion is being adjusted using the first operation, the first portionmay be adjusted using a second operation.

The resulting processed image data may be output to the display 12B forpresentation. When outputting image data to the display 12B, the imagedata may be transmitted through the device tracking primary circuitry94. The blender 124 may blend the image data with other image data.Thus, in some cases, the blender 124 may pass on the image data withoutalteration. However, the blender 124 may, when instructed, blend (e.g.,merge, combine) the image data with designated tracking data 82generated by the tracking data generator 122. The designated trackingdata 82 may be a watermark, a pattern, or the like. Some watermarks maybe a partially transparent image, text, or shape overlay to the imageframe that may be visible or imperceptible to a person. The pattern maybe a repeated watermark or shape. The designated tracking data 82 may beformed using data that, when combined with the image frame (e.g., a datamasking matrix), adjusts a brightness level of the portion of the imagedata present at a relative location of the image frame corresponding towhere the pattern is to be located. In some cases, the pattern may be ahigh contrast pattern that follows a desirable ratio between shades ofcolors or characteristics of light used as the pattern (e.g., 3:1 colorvalue ratio, 4:1 brightness of color ratio). When a high contrastpattern, the designated tracking data 82 may override a portion of theimage frame that it is overlaid on, and thus may be an image mask thatbrings a value of the image data for that portion of the image frame toan absolute lowest brightness level (e.g., 0 percent (%) brightness) orto an absolute highest brightness level (e.g., 100% brightness). Indeed,the pattern may be a black-white pattern or may include one or morecolors in the pattern.

A pulse generator 126 may track when the designated tracking data 82 isto be presented on the display 12B. When the presentation time isupcoming (e.g., one or more image frames in the future), tracked using aclock, a counter, an interrupt from the tracking data generator 122, orthe like, the pulse generator 126 may generate a notification 100 signaltransmitted to the blender 124 via delay block 130 and to the radiocontroller 98A. The delay block 130 may introduce a delay to thetransmission of the notification 100 signal equal or substantiallysimilar to a duration of time that the tracking device 62 uses toreceive the notification 100 signal, to power on the camera subsystem102 and/or the processing subsystem 104, and the like to become ready tocapture an image of the display 12B. Delaying the notification 100signal may help align the presentation of the designated tracking data82 and the image capture operations of the tracking device 62. Inresponse to receiving the delayed notification 100 signal from the delayblock 130, the blender 124 may combine the generated tracking data andthe image frame data to prepare to present the image frame with thedesignated tracking data 82.

To prepare to capture the image of the display 12B, the tracking device62 may, in response to receiving the notification 100 signal at theradio controller 98B, decode the notification 100 signal at the pulsedecoder 132 when the notification 100 signal was encoded prior totransmission between the radio controllers 98. After decoding, thetracking device 62 may transmit the notification 100 signal to thecamera shutter controller 134 and the interrupt generator 136. Inresponse to receiving the notification 100 signal, the camera shuttercontroller 134 may return or increase a supplied power to the camerasubsystem 102 and/or instruct the camera subsystem 102 to capture animage that includes the display 12B. Images captured of the trackeddevice 60 may span one or more image frames. Furthermore, in response toreceiving the notification 100 signal, the interrupt generator 136 maygenerate an interrupt command that instructs the processing subsystem104 to pause or stop a pending or ongoing operation of the processingsubsystem 104 and to prepare to receive and process the captured imagedata from the camera subsystem 102.

Sometimes the notification 100 signal may include a timestamp from thetracked device 60 that indicates a time at which the notification 100signal was generated. The tracking device 62 may reference the timestampand adjust when it operates the camera shutter controller 134 tocompensate for transmission delays incurred when transmitting thenotification 100 signal. Furthermore, the notification 100 signal mayindicate a future time or set of times during which the tracking device62 is to operate the camera.

FIGS. 10 and 11 further describe the operations of FIGS. 8 and 9 . FIG.10 is a flow diagram of a process 148 for operating the tracked device60 during a tracking operation corresponding to the tracking device 62.The process 148 may be performed by a controller of the processor corecomplex 18 of the tracked device 60, such as a display pipelinecontroller of the display processing pipeline 120, a controller of thegraphics processing pipeline 118, or the like, in response to executinginstructions stored in a tangible, non-transitory, computer-readablemedium, such as the local memory 20, the main memory storage device 22,or the like of the tracked device 60. For ease of description, theprocess 148 is described as performed by the tracked device 60 as awhole. Moreover, the operations of the process 148 are shown in aparticular order; however, some of the operations may be performed in adifferent order than what is presented.

At block 150, the tracked device 60 may determine to present trackingdata in a next image frame. The next image frame may include image datafrom the display processing pipeline 120 that is provided to the blender124. The tracking data generator 122 may generate the tracking data. Thenext image frame may be the immediately next frame to be displayed onthe electronic display of the tracked device 60 or a next image frame ina queue of image frames for subsequent presentation. The tracked device60 may determine when to present the tracking data based on an interruptbeing generated, a tracked duration of time expiring, such as from acounter counting up or down to a particular value or a timer expiring.In some examples, the tracked device 60 may monitor for an indicationsaved into a register to determine when to transmit tracking data to theblender 124. This may separate the acts of generating the tracking dataand generating the image data. That is, other graphics or image dataprocessing circuitry of the tracked device (e.g., a GPU, a displaypipeline, an image signal processor, or the like) may be agnostic as tothe tracking data. In other examples, however, generating image datathat includes the tracking data may be an operation of the graphics orimage data processing circuitry of the tracked device (e.g., a GPU, adisplay pipeline, an image signal processor, or the like).

In response to determining to present the tracking data in the nextimage frame, the tracked device 60 may generate and transmit thenotification 100 signal to the tracking device 62. After waiting aduration of time (e.g., a duration of time set by the delay block 130),the tracked device 60 may present the tracking data on the display 12Bwith the image frame to be presented. As a reminder, the tracking datais combined with the image data of the image frame to form thedesignated tracking data 82 when presented via the display 12B. Bypresenting the tracking data, the tracked device 60 may provide thetracking data to the tracking device 62 via the display 12B. After thedesignated tracking data 82 was presented on the display 12B, at block156, the tracked device 60 may proceed with subsequent imagepresentation operations.

The tracked device 60 may present the designated tracking data 82 forone or more image frame durations. To do so, the tracked device 60 mayeither continue to present the same image data, refreshing as desired.Sometimes the tracked device 60 may continue to blend incoming imageframes with the generated tracking data from the tracking data generator122 at the blender 124 for the subsequent image frames.

Considering now operations performed by the tracking device 62, FIG. 11is a flow diagram of a process 168 for operating the tracking device 62during a tracking operation to track the tracked device 60. Theoperations of the process 168 may be performed at least partially inparallel to some of the operations of process 148 of FIG. 10 . Theprocess 168 may be performed by a controller of the processor corecomplex 18 of the tracking device 62, such as the device trackingsecondary circuitry 96, a controller associated with the processingsubsystem 104 that is not power gated with the processing subsystem 104,or the like, in response to executing instructions stored in a tangible,non-transitory, computer-readable medium, such as the local memory 20,the main memory storage device 22, or the like of the tracking device62. For ease of description, the process 168 is described as performedby the tracked device 62 as a whole. The operations of the process 168are shown in a particular order; however, some of the operations may beperformed in a different order than what is presented.

At block 170, the tracking device 62 may receive the notification 100signal from the tracked device 60, and the notification 100 signal mayindicate that an incoming image frame has the designated tracking data82. In response to the notification 100 signal, at block 172, thetracking device 62 may increase a power supplied to the camera subsystem102 and/or the processing subsystem 104. The camera subsystem 102 and/orthe processing subsystem 104 may be reduced to different voltage orcurrent supply levels when idle and thus may be brought online orpowered on again using different voltages or current differentialsrelative to each other in response to the notification 100 signal. Bypresenting the tracking data, the tracked device 60 may provide thetracking data for reception by the tracking device 62 via the display12B. In some cases, the tracked device 60 may provide an indication ofthe tracking data 82 to the tracking device 62 via the notification 100signal. The tracking device 62 may store an indication of the trackingdata in its memory 20 and may use the indication to identify wherewithin a dataset the tracking data is located.

Once powered on, at block 174, the tracking device 62 may instruct thecamera subsystem 102 to take an image capture (e.g., perform imagecapturing operations) a duration of time after receiving the pulse. Theduration of time may correspond to a length of time that the delay block130 delays the notification 100 signal. Sometimes, the tracking device62 may adjust when the camera subsystem 102 is instructed to perform animage capture based on a timestamp included in the notification 100signal. Indeed, the tracking device 62 may compensate for transmissiondelays determinable by comparing a time of reception to the timestamp ofthe notification 100 signal. The notification 100 signal may indicate aspecific time at which the camera subsystem 102 is to be operated tocapture an image of the tracked device 60, which may happen when, forexample, the tracked device 60 presents images at variable refresh ratesotherwise unknown to the tracking device 62. In any of these cases, thecamera subsystem 102 captures an image of the display 12B displaying thedesignated tracking data 82 in response to the instruction.

After capturing the image data, at block 176, the tracking device 62 maydetermine a position of the display 12B of the tracked device 60 basedon the captured image data including the designated tracking data 82.Since the designated tracking data 82 is a displayed marker known bytracking device 62 and seen by cameras of the tracking device 62, thetracking device 62 may identify the designated tracking data 82 in thecaptured image data and may correlate the location of the designatedtracking data 82 in the space of the ambient environment 66 to being thelocation of the display 12B and the location of the tracked device 60.For example, the tracking device 62 may compare an indication of thedesignated tracking data 82 to the image data captured of the display12B to identify a location of the designated tracking data 82 in thecaptured image data. Being known and pre-defined, visual features of thedesignated tracking data 82 are also known by the tracking device 62,and thus such a known marker may be used to track the tracked device 60even when custom and dynamically changing image content is beingdisplayed. Furthermore, the designated tracking data 82 may be overlaidfor one or more frames, where a single frame lasts a relatively short,imperceptible amount of time (e.g., 8.33 milliseconds (ms) of time for aframe presented at a 120 frames-per-second (fps) refresh rate).Moreover, the designated tracking data 82 may be even less noticeable bya person given the blending operations of the blender 124 furtherdisguising the designated tracking data 82 as the known marker withinthe image content.

At block 178, the tracking device 62 may generate an image datamodification based on the location, may perform an operation based onthe location, or both. Indeed, after identifying the designated trackingdata 82 in the captured image data, the tracking device 62 may adjustits operations based on the location of the display 12B, such as tonotify a person to return within programmatically monitored boundariesof the ambient environment 66 (e.g., an alert to notify the user “pleasereturn to the play area of this virtual world”) and/or to generate imagedata to overlay on the captured image data, such as to manipulate spacesurrounding the tracked device 60 for the amusement of a viewer or tootherwise convey information.

The above descriptions focus on using the designated tracking data 82 asthe known marker. However, these systems and methods may also work withusing, as the known marker, a static image frame, or defined imagefeatures in presented images.

Indeed, FIG. 12 is a diagrammatic representation of the tracked device60 transmitting an image data sample 190 to the tracking device 62additionally or alternatively to the notification 100 signal. Thetracked device 60 may use the image data sample 190 similar to thenotification 100 signal, and thus the transmission of either may notifythe tracking device 62 of an incoming image frame to trigger bringingthe tracking device 62 out of a reduced power operational mode. Forcomponents in FIGS. 12-15 that have been described above, earlierdescriptions apply.

To alert the tracking device 62 of an incoming known marker, the trackeddevice 60 may transmit the image data sample 190. After receiving theimage data sample 190, the tracking device 60 may perform operations ofblocks 172-178 of FIG. 11 . These operations may include increasingpower supplied to idle circuitry (e.g., camera subsystem 102, processingsubsystem 104) to prepare to capture an image of the display 12B andanalyzing the captured image data to identify the location based on theknown marker.

The image data sample 190 may include an indication of an upcoming imageframe that the tracking device 62 is about to present as a known marker,such as a whole image frame, a partial image frame, or extractedportions of an image frame that may be identifiable when processing theimage data. The image data sample 190 may indicate extracted visualfeatures to the tracking device 62 to use when identifying the locationof the display 12B. Indeed, the tracked device 60 may transmit anindication of a static image frame as the image data sample 190 and/oran indication of defined image features in images presented or to bepresented, which may include a region of color or brightness contrast inthe image frame, a text, a shape, or any graphical feature relativelyprominent in an presented image. The tracking device 62 may output thelocation 106 of the tracked device after performing operations of block176 and may use the location 106 in other operations at block 178.

To use the image data sample 190 as the known marker, circuitry and/oroperations performed by each of the device tracking primary circuitry 94and the device tracking secondary circuitry 96 may change. For example,FIG. 13 is a block diagram of the tracked device 60 using a subsampler202 to intersect and sample a subset of image data, or image data for awhole image frame, being transmitted from the display processingpipeline 120 (e.g., the image data sample 190). The sample of the imagedata may be transmitted to the tracking device 62 to enable the trackingdevice 62 to superimpose the sample of the image data over the capturedimage data as a way to hide tracking data of the display 12B from beingpresented to the user. In some cases, however, the sample of the imagedata may be used as a known marker of image data for the tracking device62 to use when identifying the location of the display 12B. Thesubsampler 202 may transmit the image data sample 190 to a compressor204. When used, the compressor 204 compresses the image data sample 190before its transmission to the tracking device 62 as the known marker.The compressor 204 may use any suitable method, such as inter-framecoding (e.g., encoding or decoding first image data based on values ofsecond image data to reduce data handled when duplicated), intra-framecoding (e.g., encoding or decoding first image data based on values ofthe first image data to reduce data handled when duplicated), lossycompression, lossless compression, or the like.

In another example, the image data sample 190 may include extractedimage features for the tracking device 60 to use as a known marker whensearching the captured image data for the display 12B. Indeed, FIG. 14is a block diagram of the tracked device 60 using a feature extractor212 to intersect and sample a subset of image data, or image data for awhole image frame, being transmitted from the display processingpipeline 120 for identification as a known marker of image data for thetracking device 62. The feature extractor 212 may extract one or moreimage features, or subsets of image data, from the image data sample190. For example, the feature extractor 212 may determine the imagefeatures to be extracted based on relative contrast values betweendifferent portions of the image data, text to be presented in the imageframe, or the like, to identify relatively clear features readilyidentifiable in the captured image data of the presented image frame.Once extracted, the compressor 204 receives the extracted data from thefeature extractor 212 and compresses the extracted data for transmissionto the tracking device 62.

Both FIG. 13 and FIG. 14 show that the tracked device 60 to include thedelay block 130 coupled to the display 12B. In these other cases, thedelay block 130 may be programmed to have a same delay as in FIG. 9since the delay being compensated for includes a power on delay timeassociated with bringing components of the tracking device 62 backonline after being powered off or in a reduced power state when idle.The delay block 130 may add time to a total transmission time of theimage data to the display 12B, such as to display drivers or controlcircuitry of the display 12B, without adding additional the image datasample 190 beyond transmission and processing delays. Delaying outputsfrom the display processing pipeline 120 may synchronize the imagepresentation and the image capture operations.

In some cases, the tracking device 62 generates image data tosuperimpose on a portion of captured image data (e.g., captured imagedata depicting when presented the ambient environment and any devices orobjects included within). FIG. 15 is a diagrammatic representation ofthe tracking device 62 superimposing or overlaying image 224 datagenerated by its own image processing circuitry on captured image datadepicting the tracked device 60 when presented via the display 12A.Indeed, the tracking device 62 may generate image frames for conversioninto image data and may combine the generated image data with thecaptured image data. The tracking device 62 may combine the two datasetssuch that the generated image data is superimposed on the captured imagedata at a location in the dataset of designated tracking data 82determined to correspond to the location of the display 12B. Thus, to doso, as is generally described above, the tracking device 62 searches thecaptured image data to identify where in the dataset the designatedtracking data 82 is found, such as by using any suitable imageprocessing comparison operation). Once the location within the datasetis identified, the tracking device 62 performs combining operations,such as image data masking operations that use matrices to replacematrix values with new data, to replace the captured image data with thegenerated image data at the location within the dataset. Thus, when thecombined data is presented on the display 12A, an image 224 generated ofthe tracked device 60 may omit an image of designated tracking data 82even when the tracked device 60 itself is continuing to present thedesignated tracking data 82. In these cases, the tracked device 60 maycontinue presenting the designated tracking data 82 without concern ofhow noticeable the designated tracking data 82 is to a user of thetracking device 62, since the image 224 presented to the user of thetracked device 60 is filtered or adjusted to omit the designatedtracking data 82.

These systems and methods support the designated tracking data 82 beingpresented for more or less than an image frame time duration. The delayblock 130 may be calibrated accordingly to provide a suitable amount ofdelay. The designated tracking data 82 presentation duration may also beconsidered a presentation variable to adjust an appearance of thedesignated tracking data 82. Furthermore, the designated tracking data82 may be adjusted dynamically based on ambient brightness level, andthus may be presented differently when the display 12B is outside or isinside. For example, the tracked device 60 may adjust the designatedtracking data 82 to be presented using higher contrast colors or usingbrighter color values based on a sensed ambient brightness level of theambient environment 66 (e.g., ambient environmental conditions), such asby adjusting an overall screen brightness value or adjusting the imagedata used as the designated tracking data.

Keeping the foregoing in mind, some systems may use secondary or secondtracking data to validate, in place of, or with the tracking data 82.FIG. 16 is a block diagram of the tracked device 60 that generates bothfirst tracking data 82 for presentation via the display 12B (e.g., thatis received by tracking device 62 via image capture of the display 12Bby the camera subsystem 102) and second tracking data 230 fortransmission to the tracking device 62. The second tracking data 230 mayinclude any suitable electromagnetic or radio frequency (RF) waveform, amagnetic field, infrared signals, audio signals, or the like. Althoughthis example uses the tracking data 82 as first tracking data 82, itshould be understood that any combination of tracking data described asthe second tracking data 230 may be used in combination of another typeof tracking data described. For example, an audio signal may be usedwith an infrared signal as the two types of tracking data used to locatethe tracked device 60.

As discussed above, the tracked device 60 may present the tracking data82 alone or with image data of a presented image frame, and the trackingdevice 62 may take an image capture to obtain image data that includesthe tracking data 82. Image data from the image capture of the display12 is generated by the camera subsystem 102 and transmitted to theprocessing subsystem 104 for processing.

The tracked device 60 may also generate second tracking data 230 (e.g.,secondary tracking data). The tracking device 62 may use the secondtracking data 230 to validate a location of the tracked device 60determined based on the first tracking data 82. The tracking device 62may sometimes use the second tracking data 230 to determine the locationof the tracked device 60 without using the first tracking data 82. Insome cases, the tracking device 62 may toggle between operational modes,where each operational mode may use a different of the tracking data.For example, the tracking device 62 may operate in two or moreoperational modes. A first operational mode may cause the trackingdevice 62 to track the tracked device 60 using the first tracking data82 and a second operational mode may cause the tracking device 62 totrack the tracked device 60 using the second tracking data 230. As laterdescribed in FIG. 17 , the operational modes may be used to operate thetracking device 62 in and out of a relatively lower power consumptionoperational mode since tracking using the second tracking data 230 mayuse fewer processing resources than tracking using the first trackingdata 82. Toggling operation of the tracking device 62 between thedifferent operational modes may improve device operation by reducingoverall power consumption and permitting a controller to tailor theoperation of the tracking device 62 to the current operational demandsof the tracking circumstances. For example, the tracking device 62 maybe operated to use a less accurate, lower power tracking mode when thetracked device 60 is static (e.g., indicated by motion data, positiondata, or orientation data) or relatively easy to track, but may beoperated to use a higher accuracy, higher power tracking mode when thetracked device 60 is not static and/or is relatively difficult to track.The higher accuracy, higher power tracking mode may be suitable for usein bright rooms, when the display 12B is dark, or otherwise when theresulting captured image data may have relatively low contrast betweenits respective portions of image data. The higher accuracy, higher powertracking mode may also be suitable for use when the display 12B isblocked, such as by a user, another device, or the electronic device 10itself like when the electronic device 10 is flipped over and thedisplay 12B is no longer visible or able to be captured by the camerasubsystem 102 of the tracking device 62). The high power tracking modemay use the first tracking data 82 and may validate the locationdetermined using the first tracking data 82 with the second trackingdata 230.

Example operations based on the first tracking data 82 and the secondtracking data 230 are shown in FIG. 17 . FIG. 17 is a flow diagram of aprocess 240 for operating the tracking device 62 during a trackingoperation. The process 240 may be performed by a controller of theprocessor core complex 18 of the tracking device 62 in response toexecuting instructions stored in a tangible, non-transitory,computer-readable medium, such as the local memory 20, the main memorystorage device 22, or the like of the tracking device 62. For ease ofdescription, the process 240 is described as performed by the trackingdevice 62 as a whole. Moreover, the operations of the process 240 areshown in a particular order; however, some of the operations may beperformed in a different order than what is presented.

At block 242, the tracking device 62 may receive an indication of aresource consumption tracking mode. A first indication (e.g., controlsignal, flag, data) may indicate a first tracking mode and a secondindication may indicate a second tracking mode. Similar methods andsystems for indications may be used to indicate precision modes. Thus,the determination at block 242 may sometimes be one for a reducedprecision tracking mode. A lack of the first indication may indicate thesecond tracking mode. The tracking device 62 may receive an input at aninput device or touch pixel of the display 12A that generates the firstindication and/or may include control circuitry that generates the firstindication in response to determining to enter the reduced powerconsumption mode.

After receiving and/or generating the first indication, the trackingdevice 62 may, at block 244, determine whether a reduced resourceconsumption tracking mode is indicated. The reduced resource consumptiontracking mode may correspond to a first of two resource consumptiontracking modes (e.g., higher and lower consumption modes).

When indicated, the tracking device 62 may, at block 246, identify thelocation of the tracked device 60 using the second tracking data 230.Using the second tracking data 230 to determine the location may consumefewer computing or power resources relative to using image dataprocessing operations associated with processing the first tracking data82. The second tracking data 230 may include any suitable data usablefor locating or triangulating to the tracked device 60 from one or morereference points, such as the location of the tracking device 62 itself.In some cases, the tracking device 62 may correlate the location of thetracked device 62 to a direction or orientation that a relativelystrongest signal strength of the second tracking data 230 originates.Determining the location of the tracked device 60 using the secondtracking data 230 without the first tracking data 82 may consume fewerprocessing resources, and thus be performed during the reduced resourceconsumption tracking mode. As discussed below, the tracking device 62may determine that the location determined based on the second trackingdata 230 has a low confidence value in its accuracy by considering, atblock 256, whether the confidence in the location determined based onthe second tracking data 230 in greater than or equal to a thresholdamount. When the confidence value is greater than or equal to thethreshold value, the tracking device 62 determines that the confidencevalue is suitably high and, at block 254, modifies image data, performsan operation, or both, based on the location determined using the secondtracking data 230.

However, when the confidence value is less than the threshold value, thetracking device 62 may find the confidence value too low and thus mayverify the location by, at block 248, identifying the location of thetracked device 670 using the first tracking data 82. Another time wherethe first tracking data 82 may be considered is when the reducedresource mode is not indicated at block 244.

Referring back to block 244 operations, when the reduced resource modeis not indicated, the tracking device 62 may, at block 248, identify thelocation of the tracked device 60 using the first tracking data 82. Thismay involve performing operations of FIGS. 10 and 11 , and thusdescriptions are relied on herein. At block 250, the tracking device 62may determine whether a confidence amount corresponding to the locationof the tracked device 60 determined based on the first tracking databeing correct is greater than or equal to a threshold amount.

Any suitable method may be used to evaluate a confidence level at block256 and/or block 250. For example, certain conditions may trigger arelatively low confidence level. The determination may be based on apopulation mean, a standard deviation, or any other suitable statisticaldetermination to identify a confidence value in the locationdetermination. For example, when the tracked device 60 has the display12B blocked from the camera subsystem 102, the tracking device 62 maydetermine, at block 250, a confidence in the location determined basedon the first tracking data 82 is a value less than or equal to thethreshold amount, and thus may, at block 252, validate the location ofthe tracked device 60 based on the second tracking data 230.

To elaborate, in response to determining that the confidence valuecalculated for the determined location of the tracked device 60 based onthe first tracking data 82 is not greater than or equal to the thresholdamount, the tracking device 62 may validate, at block 252, thedetermined location using the second tracking data 230. For example, alocation of the tracked device 60 may be determined by comparing thesecond tracking data 230 to expected tracking data or a known referencelocation to determine a difference used to identify the location. Thelocation based on the first tracking data 82 may be compared to thelocation found based on the second tracking data 230 to validate orreject the location.

Referring back to block 256, when evaluating the confidence value forthe location determined based on the second tracking data 230, thetracking device 62 may identify, at block 248, the location of thetracked device 60 using the first tracking data 82. Although thetracking device 62 was originally instructed to operate in the reducedresource consumption tracking mode (e.g., lower power consumption), thetracking device 62 may thus use operations of block 256 to determine tofurther validate the location determined based on the second trackingdata using operations of at block 246. The determinations of block 256may permit selective enablement of the higher resource consumption modewhen higher accuracy in location or confidence levels is requested.

In response to determining that the confidence value calculated for thedetermined location of the tracked device 60 is greater than or equal tothe threshold amount, or after validating the location of the trackeddevice 60, the tracking device 62 may, at block 254, modify image data,perform an operation, or both, based on the location of the trackeddevice 60 (e.g., the location of the display 12B). Operations performedhere are similar to those of block 178 of FIG. 11 and may be based oneither the second tracking data 230, the first tracking data 82, orboth.

The tracking device 62 may use motion sensing data from an inertialmeasurement unit (IMU) of the tracked device 60, its own IMU, or both topredict where within an image frame region that the display 12B isexpected to be at a future time. Data from the IMUS may be consideredsecond tracking data 230, or secondary tracking data, as discussed inFIGS. 16-18 . Each IMU may sense velocity, acceleration, amplitudes,torques, rotations and/or changes in any of the values, to help providea numerical indication of IMU movement. Predicting future locations ofthe tracked device 60 may reduce a total time spent identifying alocation of the display 12B by reducing an amount of image data searchedby the tracking device 62 to just the regions of image data expected toinclude indications of the display 12B.

The notification 100 signal may be used in addition to a refresh pulse,not shown in the figures. The tracked device 60 may send the refreshpulse each time that an image frame presented on the display 12B isupdated, and thus may send the refresh pulse at a rate that matches arefresh rate of the display 12B. The tracking device 60 may synchronizeits camera subsystem image capturing operations to equal or besubstantially the same as the refresh rate of the display 12B inresponse to receiving each of the refresh pulses. Aligning the refreshrate of the display 12B with the image capturing operations, andsubsequent refresh rate of the display 12A, may reduce aliasing orblurring artifacts that may occur when presenting the display 12B incaptured image data via the display 12A.

The tracking device 62 and the tracked device 60 may operate to providean XR system. To do so, the tracking device 62 may transmit an initialinitiation signal that instructs the tracked device 60 to operate topresent the tracking data and/or to prepare to be tracked. Thus, thetracked device 60 may not always operate to present designated trackingdata 82. The operations described herein may correspond to anoperational mode that may be entered into and exited. The operationalmode may be entered based on whether or not the tracking device 62 isbeing operated to provide an extended reality experience (e.g., modifiedimage) presented based on the location of the tracked device 60. Thetracking device 62 may track the object 64 without also tracking thetracked device 60, and vice versa, or the tracking device 62 may trackboth the object 64 and the tracked device 60.

An XR system may provide an entertainment system. In some cases, an XRsystem may be used to provide a service, such as an object-locatingapplication that searched for tracked device 60 identification tags andgenerates graphical representations of instructions, directions, arrowsor the like to guide a user interacting with the tracking device 62toward the determined location of the tracked device 60. Otherapplications of these systems and methods may also apply although notparticularly discussed.

Either of the confidence threshold levels (e.g., threshold amount usedat block 256 and/or block 250) may be adjustable based on operation ofthe tracking device 62, application executing on the tracking device 62(e.g., accuracy-critical application or not), or the like. For example,when a first application being executed indicates a first level ofaccuracy, the tracking device 62 may adjust or increase the value ofthreshold to be more stringent, or a higher value, than the thresholdused for a second application that indicates a second level of accuracy,or does not indicate a level of accuracy.

Furthermore, these systems and methods described herein may apply beyondproprietary boundaries of hardware and/or software of electronicdevices. For example, a tracked device 60 of one company and/oroperating system may be compatible to operate in conjunction with atracking device 62 of another company or operating system. This mayinclude using proprietary wireless frequencies or characteristics. Thetracking device 62 may register to an output wireless signal from thetracked device 60 such that each may communicate at least trackingsignals to each other for the tracking device 62 to determine thelocation of the tracked device 60.

Systems and methods described herein may be used in known or unknownambient environments 66. For example, a home, primary residence, placeof work, or the like may be known ambient environments 66, which mayhave objects within the known ambient environment 66 with fixedlocations that are known to the tracking device 62. The tracking device62 may undergo a calibration, orientation, or otherwise initializingoperation to “know” or learn an ambient environment 66. Theseinitializing operations may occur over time through repeated use of thetracking device 62 in a respective ambient environment 66 In theseenvironments with objects at known, fixed locations, the tracking device62 may use a tiered system to guide a user toward the tracked device 60and/or or to track the tracked device 60. When searching for the trackeddevice 60, the tracking device 62 may use the second tracking data 230to instruct a user to move it closer to the tracked device 60 at arelatively high granularity. As the tracking device 62 gets closer tothe tracked device 60, the tracking device 62 may switch to locating thetracked device 60 using the first tracking data 82 as a way to reducegranularity and make the locating operation more accurate. For example,the tracking device 62 may originally be in located in a right side of aroom and may use a locating operation based on the second tracking data230 to determine whether the tracked device 60 is also in the right sideof the room, or is in the left side of the room, or is in a differentroom. Based on its determination of relative location of the trackeddevice 60 to the room, the tracking device 62 may generate instructionsto guide a movement of the tracking device 62 toward signals beinggenerated by the tracked device 60. When a threshold distance from thetracked device 60, when signal strength of the second tracking data isgreater than a threshold strength value), or in response to a differentcondition, the tracking device 62 may switch to locating the trackeddevice 60 using the first tracking data 82, and then may use morespecific generated graphics to indicate the location of the trackeddevice 60 and/or to track the location of the tracked device 60 (e.g.,generated graphics surrounding the display 12B of the tracked device 60in the virtual space)

Thus, the technical effects of the present disclosure include systemsand methods that improve tracking operations of a tracking device byoperating the tracking device to monitor and track known markerspresented by a tracked device. The tracking device may be part of an XRsystem that presents an enhancement, a modification, an animation or thelike in association with the tracked device based on the trackedlocation of the known markers. By providing the tracking device with aknown marker to track on the tracked device, the tracking device maytrack an electronic display of the tracked device, the known marker, thetracked device, or any combination thereof, with increased accuracy andreduced perceivable delays than otherwise experienced when trying totrack a display used to present rapidly changing or variable image data.When tracking operations based on known markers presented via a displayof the tracked device are performed continuously, mixed realityaugmentations will “stick” very smoothly on top or around, even whilethe tracked device changes images presented via the display. To furtherimprove images presented via the tracking device, the tracked device maytransmit periodic pulses at a rate that matches or is substantiallysimilar to a refresh rate of its display such that the tracking devicemay align its refresh rate with that of the tracked device. Doing so mayreduce flashing or other perceivable image artifacts generated in imagesof the display presented using an unaligned refresh rate.

The specific embodiments described above have been shown by way ofexample, and it should be understood that these embodiments may besusceptible to various modifications and alternative forms. It should befurther understood that the claims are not intended to be limited to theparticular forms disclosed, but rather to cover all modifications,equivalents, and alternatives falling within the spirit and scope ofthis disclosure.

Furthermore, it is well understood that the use of personallyidentifiable information should follow privacy policies and practicesthat are generally recognized as meeting or exceeding industry orgovernmental requirements for maintaining the privacy of users. Inparticular, personally identifiable information data should be managedand handled so as to minimize risks of unintentional or unauthorizedaccess or use, and the nature of authorized use should be clearlyindicated to users.

The techniques presented and claimed herein are referenced and appliedto material objects and concrete examples of a practical nature thatdemonstrably improve the present technical field and, as such, are notabstract, intangible or purely theoretical. Further, if any claimsappended to the end of this specification contain one or more elementsdesignated as “means for [perform]ing [a function] . . . ” or “step for[perform]ing [a function] . . . ”, it is intended that such elements areto be interpreted under 35 U.S.C. 112(f). However, for any claimscontaining elements designated in any other manner, it is intended thatsuch elements are not to be interpreted under 35 U.S.C. 112(f).

What is claimed is:
 1. A system comprising: a camera subsystemconfigured to capture an image of a first display; and processingcircuitry configured to: receive image data from the camera subsystemcorresponding to the image; detect conveyance of tracking data on thefirst display; identify a location of the tracking data within the imagedata based at least in part on comparing the image data with anindication of the tracking data; and identify a physical location of thefirst display based at least in part on the location of the trackingdata.
 2. The system of claim 1, wherein the processing circuitry isfurther configured to: store an indication of the physical location ofthe first display in a memory; and generate an alert based at least inpart on the physical location of the first display and whether thephysical location of the first display is outside a boundary, or both.3. The system of claim 1, comprising a second display, wherein theprocessing circuitry is further configured to: modify the image databased at least in part on the physical location of the first display;generate additional image data perceivable as tracking the physicallocation of the first display; and convey the modified image data, theadditional image data, or both on the second display.
 4. The system ofclaim 1, wherein the processing circuitry is further configured toreceive a pulse signal a duration of time before the camera subsystemcaptures the image.
 5. The system of claim 4, wherein the processingcircuitry, in response to receiving the pulse signal, is furtherconfigured to increase an amount of power supplied to the camerasubsystem and to a processing subsystem.
 6. The system of claim 1,wherein the processing circuitry, in response to determining passage ofa threshold duration of time from a previous tracking data presentationtime, is configured to increase an amount of power supplied to thecamera subsystem.
 7. The system of claim 1, comprising an extendedreality headset that comprises the camera subsystem and the processingcircuitry.
 8. The system of claim 1, comprising a tablet device thatcomprises the camera subsystem and the processing circuitry, wherein theprocessing circuitry is configured to execute an extended realityapplication to generate an animation or manipulate an image stream basedat least in part on a plurality of indications of a plurality ofphysical locations of the first display over time.
 9. The system ofclaim 1, wherein the tracking data comprises extracted visual featuresthat exceed a threshold amount of contrast relative to visual featuresof different portions of additional image data.
 10. A tangible,non-transitory, machine-readable medium, comprising machine-readableinstructions, configured to cause a processor of a tracking device toperform operations comprising: receiving a pulse signal from a trackeddevice, wherein the pulse signal indicates that an image frame to bepresented by the tracked device comprises tracking data; in response tothe pulse signal, increasing an amount of power supplied to a camerasubsystem; transmitting a control signal to the camera subsystem toinitiate an image capture operation while the image frame is presentedby the tracked device; receiving image data from the camera subsystemgenerated at least in part by the image capture operation; identifying alocation of the tracking data within the image data based at least inpart on a comparison between the image data and a stored indication ofthe tracking data; and performing an operation based at least in part onthe location of the tracking data within the image data.
 11. Themachine-readable medium of claim 10, wherein the operations comprise:determining that the camera subsystem is idle; and reducing the amountof power supplied to the camera subsystem in response to determiningthat the camera subsystem is idle.
 12. The machine-readable medium ofclaim 10, wherein performing the operation comprises the processorperforming operations comprising: identifying a physical location of thetracked device based at least in part on the location of the trackingdata within the image data; and generating an image data modificationbased at least in part on the physical location of the tracked device,wherein the image data modification is part of an animation perceivableas tracking the physical location of the tracked device when presented.13. The machine-readable medium of claim 10, wherein the operationscomprise: in response to the pulse signal, increasing an amount of apower supplied to a processing subsystem configured to process the imagedata as part of operations to identify the location of the tracking datawithin the image data, wherein the amount of power supplied to theprocessing subsystem is different from the amount of power supplied tothe camera subsystem after increasing both.
 14. The machine-readablemedium of claim 10, wherein transmitting the control signal to thecamera subsystem to initiate the image capture operation comprises theprocessor performing operations comprising: starting a timer in responseto receiving the pulse signal from the tracked device; and in responseto the time expiring, transmitting the control signal to the camerasubsystem to initiate the image capture operation.
 15. A device,comprising: a camera subsystem configured to capture image data at leastin part by capturing an image of a first display; and device trackingcircuitry coupled to the camera subsystem, wherein the device trackingcircuitry is configured to: increase an amount of power supplied to thecamera subsystem to cause the camera subsystem to generate the imagedata; receive the image data from the camera subsystem corresponding tothe image; identify a location of tracking data within the image databased at least in part on comparing the image data with an indication ofthe tracking data; and identify a physical location of the first displaybased at least in part on the location of the tracking data.
 16. Thedevice of claim 15, comprising a second display, wherein the devicetracking circuitry is configured to: modify the image data based atleast in part on the physical location of the first display; and conveythe modified image data on the second display.
 17. The device of claim16, wherein the device tracking circuitry is configured to filter outthe tracking data from the image data before modifying the image databased at least in part on the physical location of the first display.18. The device of claim 15, wherein the camera subsystem is configuredto capture the image data in response to receiving a control signal, andwherein the device tracking circuitry is configured to generate thecontrol signal in response to receiving a notification indicating thatthe first display is preparing to present the tracking data.
 19. Thedevice of claim 18, wherein the notification comprises a timestamp, andwherein the device tracking circuitry is configured to generate thecontrol signal at a time that compensates for a transmission delayincurred during transmission of the notification.
 20. The device ofclaim 15, wherein the indication of the tracking data comprises a copyof a subset of image data presented as the tracking data.
 21. Atangible, non-transitory, machine-readable medium, comprisingmachine-readable instructions, configured to cause a processor of atracking device to perform operations comprising: receiving firsttracking data provided by a tracked device; receiving second trackingdata provided by the tracked device; determining a location of thetracked device based on the first tracking data; determining aconfidence value for the determination of the location; in response tothe confidence value being less than or equal to a threshold confidencevalue, validating the location of the tracked device based on the secondtracking data; and performing an operation based at least in part on thevalidated location of the tracked device.
 22. The machine-readablemedium of claim 21, wherein performing the operation comprises:generating an image data modification based at least in part on thevalidated location of the tracked device, wherein the image datamodification is part of an animation perceivable as tracking the trackeddevice when presented; and presenting, via a display, the image datamodification.
 23. The machine-readable medium of claim 21, wherein thefirst tracking data comprises captured image data generated by a camerasubsystem, and wherein the second tracking data comprises an infraredsignal.
 24. The machine-readable medium of claim 21, wherein the firsttracking data comprises captured image data generated by a camerasubsystem, and wherein the second tracking data comprises a signalcorresponding to a radio frequency signal.
 25. The machine-readablemedium of claim 21, wherein the tracked device is configured toobfuscate an image frame when presenting first image data of the imageframe, wherein the first tracking data comprises second image data ofthe obfuscated image frame, and wherein performing the operationcomprises: generating the first image data based at least in part on thevalidated location of the tracked device and the second image data; andpresenting, via a second display of the tracking device, the first imagedata.